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Editor's  Note:  Normally  papers  were  written  after  the  authors 
have  gotten  at  least  some  results  to  prove  a  promt.  Not  so  for 
the  invited  article  below!  In  this  case  the  authors  are  charged 
with  managing  a  brand  new  concept,  almost  like  a  dream  and 
turning  it  into  reality  some  day.  Here  is  how  they  describe  their 


plans. 


Robotic  telepresence:  Applications  of  human  controlled 
robots  in  Air  Force  maintenance 


ABSTRACT 
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""This  paper  describes  the  major  technology  required  to  perform 
human-controlled  robotic  servicing  of  military  aircraft  during  sortie 
regeneration  in  a  chemical,  radiological,  or  biological  attack.  Current  and 
projected  Air  Force  aircraft  are  designed  to  be  serviced  by  human  maintenance 
personnel.  Wartime  operations  might  make  it  desirable  to  remove  these 
maintenance  personnel  from  the  hazardous  flight  line  environment.  Robots  with 
human-like  interface  characteristics  such  as  dexterous  manipulators,  vision, 
touch  and  hearing  might  be  necessary  to  complete  these  tasks.  The 
presentation  will  outline  "Robotic  Telepresence"  as  one  solution  to  the 
complex  requirement  of  rearming  and  servicing  aircraft  without  exposing 
humans.  The  technology,  when  fully  developed,  will  have  maintenance  personnel 
control  robots  that  have  quasi -anthropoid  features  from  remote  stations. 

Unique  human  capabilities  are  to  be  retained  in  the  control  loop  while  robotic 
ruggedness  will  be  exploited.  Research  and  development  opportunities  cross 
the  traditional  boundaries  between  hardware  engineering  and  human  factors 
applications.  Unique  challenges  in  this  approach  to  integrate  the  human  and 
the  machine  will  be  identified. 
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INTRODUCTION 


Mission  essential  operations  must  be  maintained  in  all  military 
operational  scenarios.  Current  operational  capability  is  threatened  by  the 
degradation  of  flight  line  crew  performance  in  Nuclear,  Biological  and/or 
Chemical  (NBC)  environments.  These  environments  may  lead  to  total  crew 
incapacitation  and  total  loss  of  mission  capabilities.  One  potential  concept 
for  maintaining  mission  effectiveness  in  the  NBC  contaminated  environment  is 
Robotic  Telepresence  (R/T).  This  recently  initiated  concept  would  replace 
exposed  maintenance  personnel  with  robots  that  are  under  the  direct,  real-time 
control  of  remote  human  operators  (Mohr,  1986).  Robotic  Telepresence  would 
allow  the  operators  to  "see,  feel  and  hear"  the  aircraft  servicing  environment 
through  sensory  systems  on  the  robot.  These  human  controlled  robots  could  be 
used  in  various  flight  line  aircraft  servicing  scenarios  such  as  rearming 
weapons  systems  and  refueling  aircraft.  Other  applications  include  minor 
repair  such  as  "black  box"  removal  and  replacement  (Figure  1).  Additional 
uses  in  tire  and  wheel  changing  and  battle  damage  assessment  may  be  developed. 
The  concept  integrates  the  human's  ability  to  work  in  the  unstructured  flight 
line  environment  with  the  robot's  ability  to  work  in  lethal  environments 
contaminated  by  NBC  weapons.  Human-in-the-lcop  control  of  flight  line  robots 
may  one  day  play  a  critical  role  in  mainlining  and  enhancing  mission 
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effectiveness  in  these  environments  by  reducing  human  exposure  and  crew 
incapacitation  in  the  NBC  environments.  In  addition,  a  mature  R/T  technology 
might  also  achieve  force  multiplication  via  a  one-to-many  mapping  between 
human  operator  and  R/T  robot. 

MAINTENANCE  ENVIRONMENT 

Wartime  aircraft  maintenance  operations  may  require  personnel  to  perform 
servicing  functions  on  flight  lines  that  have  been  contaminated  with  disabling 
or  lethal  doses  of  NBC  agents.  These  personnel  must  wear  bulky  protective 
garments  that  substantially  reduce  effective  task  performance  by  limiting  or 
hindering  communications,  reducing  fine  manipulative  dexterity,  and  decreasing 
operational  time  at  the  work  site  due  to  heat  stress  and  fatigue  factors.  As 
a  consequence,  required  aircraft  servicing  may  be  interrupted,  delayed  or  even 
prevented  by  these  factors. 

Although  the  servicing  tasks  which  must  be  performed  in  the  NBC 
environments  are  practiced  continually  by  maintenance  crews,  they  are 
essentially  unstructured.  The  servicing  operations  were  designed  to  be 
completed  by  humans  who  can  inspect,  evaluate  and  manipulate  objects  within 
the  environment.  The  work  is  unstructured  in  that  placements  of  hardware  are 
inexact,  aircraft  orientation  and  attitude  may  change  during  the 
loading/refueling  operations  and  the  precise  operations  to  be  completed  will 
vary  from  aircraft  to  aircraft  from  day  to  day.  Consequently,  task 
completion  places  heavy  emphasis  on  human  recognition  of  the  work  site  "as  it 
is"  before  work  is  started,  human  understanding  of  the  work  site  "as  it  should 
be  at  completion",  human  assessment  of  "what  needs  to  be  dors"  and  appropriate 
real-time  adjustments  to  successfully  execute  the  operation.  Robotic 
Telepresence  will  project  these  unstructured  work  sites  to  the  human  operator 
to  allow  the  intelligent,  timely  decisions  required  for  successful  completion 
of  the  servicing  tasks. 

ROBOTIC  TELEPRESENCE 

In  order  to  gain  an  appreciation  for  human-in-the-loop  control  of  robots, 
an  understanding  of  standard  autonomous  robot  operation  is  helpful.  Figure  2 
shows  a  typical  control  circuit  for  an  industrial  robot.  The  robot  is 
programmed  to  do  a  specific  task  ir.  a  specific  environment  with  specific 
hardware.  As  the  task  proceeds  and  the  robot  interacts  with  the  work 
environment,  the  robot  state  and  the  task  state  are  monitored  by  the  low  level 
controller  which  provides  the  robot  with  feedback  control  signals  based  on 
predefined  parameters.  When  the  predefined  parameters  are  exceeded,  the 
system  should  gracefully  shut  down.  Note  that  there  is  no  interactive  control 
information  feedback  to  the  task  programming  block  once  the  high  level 
instructions  are  issued.  The  low  level  controller  communicates  with  the  task 
programmer  only  at  predetermined  intervals. 

One  potential  control  configuration  for  Robotic  Telepresence  is  shown  in 
Figure  3.  This  configuration  presents  the  robot/task  state  information  to  the 
human  operator  who  is  now  aware  of  the  real-time  status  of  the  robot/task. 

This  awareness  permits  real-time  changes  in  the  instruction  set  to  the 
computer  which  in  turn  provides  the  low  level  control  to  the  robot.  The 
robot/task  state  information  is  also  beck  to  the  low  level  controller  to 

provide  control  loop  stability  and  to  integrate  the  status  with  the  next  set 
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of  operator  instructions.  The  human  operator  can  now  "interact"  with  the  work 
environment  to  adjust  to  unanticipated  changes  in  the  unstructured  workload. 

The  human  has  several  methods  of  natural  feedback  as  shown  in  Figure  4. 
Robotic  Telepresence  seeks  to  use  as  many  of  these  feedback  systems  as  needed 
to  achieve  synergism  between  human  and  robot.  Visual,  audio,  tactile, 
kinematic,  propriocepti ve,  and  vestibular  sensors  are  all  used  by  humans  to 
sense  environmental  information  for  cognitive  processing.  Although  vision 
processes  are  heavily  involved  in  much  of  the  information  gathering,  the  other 
senses  are  capable  of  independently  supplying  critical  data  about  the 
interaction  between  the  human  and  his/her  world.  In  fact,  the  loss  of  vision 
or  any  other  single  sense  does  not  render  the  human  totally  nonfunctional. 
Similarly,  the  manipulation  of  objects  by  human  hands  does  not  require  visual 
sensory  input,  but  visual  information  about  size  and  anticipated  force/torque 
requirements  certainly  enhances  object  manipulation.  Proprioceptive  data  can 
be  used  to  gauge  size  and  shape  while  yielding  very  little  tactile  information 
about  the  texture  and  temperature  of  an  object.  Audio  feedback  is  another 
separate  channel  that  can  provide  specific  information  about  the  work  site. 
Smith  (personal  communication)  indicates  that  operators  of  the  Naval  Ocean 
Systems  Center's  remotely  operated  dune  buggy  found  information  from  the  audio 
channels  to  be  very  beneficial  relative  to  feedback  about  engine  loading  and 
as  a  surveillance  method.  Robotic  Telepresence  seeks  to  exploit  these 
multiple  paths  of  human  information  collection  by  sensing  information  at  the 
robot  work  site  and  feeding  the  information  back  to  the  human  operator  in  a 
manner  that  is  consistent  with  "normal"  direct  human  perception. 

AREAS  OF  MAJOR  WORK 
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Since  the  Air  Force  program  in  human  controlled  robots  is  still  in  its 
embroytic  stages,  the  research  to  develop  Robotic  Telepresence  and  synergism 
between  the  human  and  the  robot  will  be  initiated  in  several  key  areas.  The 
initial  focus  will  be  to  investigate  the  applicability  and  efficiency  of 
feedback  mechanisms.  The  emphasis  will  be  to  develop  the  generic  methods  to 
reflect  information  sensed  at  the  work  site  back  to  the  human  operator  at  the 
control  room.  The  goal  is  to  present  this  sensory  signal  to  the  operator  in  a 
naturally  perceptible  manner  so  that  the  requirement  for  human  integration  of 
unnatural  feedback  is  minimized.  An  example  of  unnatural  feedback  could  be  end 
effector  force/torque  data  displayed  as  a  series  of  bar  graphs  which  the  human 
must  assemble  into  a  mental  image  of  the  force/torque  vectors.  Experiments  on 
feedback  mechanisms  are  planned  in  the  following  major  areas:  (a)  vision;  (b) 
fine  dexterous  manipulations;  (c)  coarse  dexterous  manipulations;  (d)  force 
reflection  mechanisms;  (e)  tactile  sensing  and  stimulation  and  (f)  auditory 
localization.  While  the  experiments  and  developmental  research  in  tiiese  areas 
will  include  hardware  and  software  development,  the  focus  will  center  on 
developing  the  capability  to  complete  tasks  by  human  remote  control  of  robots. 

Since  visual  observation  is  a  primary  human  method  of  interacting  with 
the  environment,  a  high  quality  visual  display  system  will  be  one  key  element 
in  Robotic  Telepresence.  Present  concepts  for  visual  displays  include 
wide-field-of-view,  binocular,  helmet  mounted  systems  with  t.hp  ability  tc 
project  three  dimc..->ional  imagery  from  stereo  cameras  mounted  on  the  robot. 
Several  initial  factors  must  be  considered  and  investigated  in  the  vision 
concepts.  The  advantages  and/or  disadvantages  of  color  vision  must  be 
characterized.  The  effects  of  stereoscopic  vs  2D  flat  panels  must  be 
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evaluated.  Eye  slew  rates  and  tracking  must  be  quantified  and  integrated  into 
the  display  system  in  order  to  keep  the  operator  from  experiencing  tracking 
delays.  These  and  other  factors  associated  with  displaying  high  quality  video 
information  to  the  human  operator  must  be  examined  in  the  development  of  the 
Robotic  Telepresence  visual  display  system. 

Dexterous  manipulation  is  key  component  in  the  R/T  concept  that  can  be 
viewed  from  at  least  two  perspectives:  fine  manipulation  with  the  hand  and 
gross  manipulation  with  the  arms.  R/T  manipulation  research  will  be  initially 
dichotomized  into  these  two  subgroups.  Our  fine  manipulation  studies  will 
concentrate  on  the  control  of  a  left-right  pair  of  Utah-MIT  Dexterous  Hands 
(Jacobson,  1986)  using  hand  exoskeletons  (positional  masters)  on  the 
operator's  hands.  An  initial  area  of  investigation  will  examine  the  number  of 
degrees  of  freedom  (DOF)  required  to  perform  various  generic  and  operational 
manipulation  tasks.  Since  the  Utah-MIT  Dexterous  Hands  have  the  capability  of 
disabling  one  or  more  of  its  16  individual  joints,  they  can  be  used  to 
investigate  central  methods  for  gripping  and  to  evaluate  robotic  dexterity 
requirements.  Additional  work  is  planned  to  investigate  two  hand  operation 
and  manipulation  which  includes  interactive  work  between  the  hands  (one  hand 
exerting  forces  and  torques  on  an  object  while  the  other  hand  grasps  and 
steadies  the  object).  The  hands  have  provisions  for  tactile  sensing  pads  and 
will  be  used  as  the  platform  for  tactile  sensors  which  support  tactile 
stimulation  research.  Initial  coarse  manipulation  studies  will  be  conducted 
using  a  dual-arm  14  DOF  exoskeleton  controlling  two  6  DOF  industrial  arms. 
Areas  of  initial  investigation  will  center  on  the  14  DOF  to  12  DOF  mapping  of 
master  to  slave,  the  fidelity  of  control  required,  motion  scaling,  and 
performance  requirements. 

Force  reflection  has  been  shown  to  aid  task  performance  by  reducing  task 
completion  time  and  by  reducing  total  task  errors  (Draper,  1987).  Research  in 
force  reflection  has  been  initiated  through  a  project  to  build  a  dual-arm 
anthropomorphic  force  reflecting  excskeleton.  Upon  completion  of  this 
exoskeletal  system,  it  will  be  the  primary  research  tool  to  define  and  examine 
the  parameters  of  force  reflection  for  operational  tasks,  force  scaling,  and 
force  reflection  fidelity  requirements. 

Tactile  sensing  is  an  important  R/T  component  associated  with  the  above 
mentioned  dexterous  hands.  Tactile  sensing  is  the  first  half  of  the  challenge 
of  getting  work  site  information  back  to  the  human  operator.  Tactile 
stimulation  of  the  operator's  hands  must  reflect  "tactile  sensing"  at  the 
robot's  hands  to  complete  the  tactile  feedback  loop.  Combination  of  tactile 
stimulation  with  hand  force  reflection  will  be  investigated  to  identify  the 
central  parameters  needed  to  recognize  shapes,  forms  and  textures  independent 
of  visual  input.  Although  technology  exists  to  build  workable  tactile 
stimulators  such  as  the  OPTACON  (Bliss,  1970),  a  flexible  tactile  stimulator 
that  fits  in  the  operator's  gluve  is  still  being  pursued  (Terwil liger,  1984; 
Foley,  1987).  While  commercial  tactile/touch  sensors  are  available  from 
various  manufacturers,  the  flexible,  miniaturized  tactile  sensor  arrays 
reauired  to  detect  subtle  detail  at  the  wo»k  site  are  nut  yei.  within  the  s+:,te 
of  the  art. 

As  previously  mentioned,  the  auditory  channel  provides  additional 
information  to  the  R/T  operator.  While  the  role  of  auditory  feedback  beyond 
communi cation  and  warning  has  yet  to  be  completely  defined,  this  feedback  mode 


will  supplement  the  other  human  feedback  channels  to  permit  more  efficient 
operation  of  the  R/T  system.  Although  current  auditory  localization  research 
is  not  directed  at  R/T  (Doll,  1985),  progress  will  be  followed  closely  for 
potential  application. 


SIMULATION  OPPORTUNITIES 

Simulation  has  a  strong  role  in  Robotic  Telepresence  development. 

Modeling  and  simulation  requirements  range  from  full  system  modeling  down  to 
individual  controller-to- joint  simulation. 

Since  R/T  is  a  human-in-the-loop  based  control  system,  direct  simulation 
response  of  a  robotic  model  to  a  human  operator's  positional  controller  is 
required.  Additional  modeling  is  needed  to  simulate  the  robot  model's 
interaction  with  its  work  site.  This  robot-to-work  site  model  could  calculate 
the  forces  and  torques  expected  due  to  the  interaction.  These  forces  and 
torques  would  then  be  used  to  drive  force  reflection  actuators  mounted  on  the 
human's  positional  hardware.  These  major  simulation  tasks  would  permit  a 
human  operator  to  interactively  control  a  simulated  robot  in  its  work  site 
model . 

At  a  lower  level,  interactive  simulations  will  be  used  to  evaluate  and 
validate  design  concepts  of  individual  joints  of  exoskeletons  and  other  force 
reflecting  positional  controllers.  These  simulations  and  models  will  be  used 
to  test  new  control  strategies,  evaluate  systems  models  and  define  key 
performance  parameters  (such  as  time  delay)  to  guide  prototype  design  and 
fabrication.  From  an  operational  perspective,  models  of  the  controller  and 
the  robot  could  be  used  to  "individual ize"  the  human  operator's  physical 
characteristics  to  the  generic  controller  and  its  associated  robot. 

SUMMARY 

Human-in-the-loop  control  of  robot  workers  has  substantial  potential  tc 
protect  maintenance  personnel  against  exposure  to  hostile  environments. 

Presently  the  Air  Force  Robotic  Telepresence  Program  is  investigating  the 
human  machine  interface  required  for  sensing  human  position  to  control  the 
robot  and  is  developing  feedback  mechanisms  to  convey  work  site  information 
back  to  the  human  operator  in  a  naturally  perceptible  manner.  Research 
efforts  have  been  initiated  in  the  areas  of  force  feedback,  tactile 
stimulation  and  visual  feedback  mechanisms.  Simulation  to  develop  interactive 
models  for  human  control  is  also  underway.  Simulation  can  effectively  reduce 
development  costs  while  shortening  overall  research  schedules. 
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